Item recommendations based on image feature data

ABSTRACT

An apparatus and method to adjust item recommendations are disclosed herein. A first image attribute of a query image is compared to a second image attribute of each of a plurality of inventory images of a plurality of inventory items to identify the inventory items similar to the query image. Item recommendations comprising the identified inventory items in a first listing order are provided for display at a remote device. A second listing order of the identified inventory items is determined based on a user preference for a particular one of the identified inventory items. At least the second listing order is provided to the remote device for re-display of the item recommendations in accordance with the second listing order.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/936,052, filed Nov. 9, 2015, which is a continuation of U.S. patentapplication Ser. No. 14/218,704, filed Mar. 18, 2014, now issued as U.S.Pat. No. 9,213,991, which is a continuation of U.S. patent applicationSer. No. 13/631,849 filed Sep. 28, 2012, now issued as U.S. Pat. No.8,737,729, which claims priority to U.S. Provisional Patent ApplicationNos. 61/541,970 filed Sep. 30, 2011; 61/554,890 filed Nov. 2, 2011; and61/567,050 filed Dec. 5, 2011, the contents of which are incorporatedherein by reference in their entirety.

TECHNICAL FIELD

The present invention relates generally to image recognition and uses ofimage data obtained from image recognition.

BACKGROUND

Images can be used to convey information more efficiently or in a waynot possible with text, particularly from the viewpoint of a userviewing the images or to facilitate electronic commerce (c-commerce).However, in order to use images based on the wealth of informationcontained therein, image processing is performed to extract, identify,or otherwise recognize attributes of the images. Once extracted, theimage data can be used in a variety of applications. Depending on theparticular application(s), certain types of image processing may beimplemented over others. In some instances, the present image processingalgorithms may be insufficient.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings) will be provided by the Office upon request and paymentof the necessary fee. Some embodiments are illustrated by way of exampleand not limitations in the figures of the accompanying drawings, inwhich:

FIG. 1 illustrates a network diagram depicting an example system forperforming image processing and using image feature data obtained fromimage processing according to some embodiments.

FIG. 2 illustrates a block diagram showing components provided withinthe system of FIG. 1 according to some embodiments.

FIG. 3 illustrates a block diagram showing image processing and imagedata usage functionalities/operations implemented in modules andlibraries/data structures/databases according to some embodiments.

FIG. 4 illustrates an example flow diagram for image processing anddetermination of image matches implemented by the modules of FIG. 3according to some embodiments.

FIG. 5A illustrates sample mean of red green blue (RGB) images as wellas that of sample means of masks after background removal for each of aplurality of dress images.

FIGS. 5B-5C illustrate clothing images and corresponding sampling masksaccording to some embodiments.

FIG. 5D illustrates an example color histogram for a given clothing itemimage according to some embodiments.

FIGS. 5E-5F illustrate HSV, LUV, YC_(b)C_(r), and Lab color spacesaccording to some embodiments.

FIG. 5G illustrates use of bins along the axes of HSV color space forimage sampling according to some embodiments.

FIG. 5H illustrates an example stacked 1D histogram corresponding to aninventory image of a multi-colored dress according to some embodiments.

FIG. 5I illustrates processing of an image to detect a pattern accordingto some embodiments.

FIG. 5J illustrates processing of an image to generate an orientationhistogram according to some embodiments.

FIG. 5K illustrates an example dominant color determination using acolor space plot according to some embodiments.

FIGS. 6-8 illustrate additional details of the flow diagram of FIG. 4according to some embodiments.

FIG. 9A illustrates a high-level block diagram of the matching phaseaccording to some embodiments.

FIG. 9B illustrates user interface (UI) screens relating to the matchingphase according to some embodiments.

FIG. 10 illustrates an example flow diagram for performing re-rankingoperations according to some embodiments.

FIG. 11 illustrates an example flow diagram relating to establishingrecommendation schemes in order to provide complementary resultsaccording to some embodiments.

FIG. 12 illustrates an example (portion of the)recommendation/complementary results in response to a query imageaccording to some embodiments.

FIG. 13 illustrates an example flow diagram for recommending inventoryitems offered by an e-commerce or online marketplace at websites notassociated with the inventory items' e-commerce or online marketplaceaccording to some embodiments.

FIGS. 14A-14E illustrate example UI screens relating to implementationof the flow diagram of FIG. 13 according to some embodiments.

FIG. 15 illustrates an example flow diagram showing operations performedby the networked system to calculate and provide recommendation data inconnection with FIGS. 13 and 14A-14E.

FIG. 16 illustrates a diagrammatic representation of a machine in theexample form of a computer system within which a set of instructions,for causing the machine to perform any one or more of the methodologiesof FIGS. 4, 6-8, 10-11, 13, and 15 according to some embodiments.

The headings provided herein are for convenience only and do notnecessarily affect the scope or meaning of the terms used.

DETAILED DESCRIPTION

Described in detail herein is an apparatus and method for extractingimage feature data from an input image and various uses of the imagefeature data. Image feature data includes, but is not limited to, colorhistogram, pattern identification, dominant color, and/or orientationhistogram corresponding to the input image. The input image comprises aphoto taken by the user to capture at least a color and/or pattern, oran image included in a website or web page not affiliated with thee-commerce site or online marketplace. The extracted image feature datais used to provide recommendations of items offered for sale at ane-commerce site or online marketplace that match and/or complement theinput image. The confidence level of the image feature data for theinput image determines the particular type of image feature data of theinventory items to compare against. In some embodiments, user preferenceindications during viewing of the (initial) recommendation resultsfacilitates re-ordering the item listing order of the recommendationresults to better meet the user's actual interested feature(s) withinthe input image. In some embodiments, one or more sources are used toobtain current fashion trends and/or color combination data (e.g.,purchasing behavior data, expert knowledge data, social network data).The fashion trends and/or color combination data are distilled into oneor more complementary rules for use in determining complementary itemsto the input image.

Various modifications to the example embodiments will be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other embodiments and applications withoutdeparting from the scope of the invention. Moreover, in the followingdescription, numerous details are set forth for the purpose ofexplanation. However, one of ordinary skill in the art will realize thatthe invention may be practiced without the use of these specificdetails. In other instances, well-known structures and processes are notshown in block diagram form in order not to obscure the description ofthe invention with unnecessary detail. Thus, the present disclosure isnot intended to be limited to the embodiments shown, but is to beaccorded the widest scope consistent with the principles and featuresdisclosed herein.

FIG. 1 illustrates a network diagram depicting an example system 100 forperforming image processing and using image feature data obtained fromimage processing according to some embodiments. A networked system 102forms a network-based publication system that provides server-sidefunctionality, via a network 104 (e.g., the Internet or Wide AreaNetwork (WAN)), to one or more clients and devices. FIG. 1 furtherillustrates, for example, one or both of a web client 106 (e.g., a webbrowser) and a programmatic client 108 executing on device machines 110and 112. In one embodiment, the publication system 100 comprises amarketplace system. In another embodiment, the publication system 100comprises other types of systems such as, but not limited to, a socialnetworking system, a matching system, a recommendation system, anelectronic commerce (e-commerce) system, and the like.

Each of the device machines 110, 112 comprises a computing device thatincludes at least a display and communication capabilities with thenetwork 104 to access the networked system 102. The device machines 110,112 comprise, but are not limited to, remote devices, work stations,computers, general purpose computers, Internet appliances, hand-helddevices, wireless devices, portable devices, wearable computers,cellular or mobile phones, portable digital assistants (PDAs), smartphones, tablets, ultrabooks, netbooks, laptops, desktops,multi-processor systems, microprocessor-based or programmable consumerelectronics, game consoles, set-top boxes, network PCs, mini-computers,and the like. Each of the device machines 110, 112 may connect with thenetwork 104 via a wired or wireless connection. For example, one or moreportions of network 104 may be an ad hoc network, an intranet, anextranet, a virtual private network (VPN), a local area network (LAN), awireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), ametropolitan area network (MAN), a portion of the Internet, a portion ofthe Public Switched Telephone Network (PSTN), a cellular telephonenetwork, a wireless network, a WiFi network, a WiMax network, anothertype of network, or a combination of two or more such networks.

Each of the device machines 110, 112 includes one or more applications(also referred to as “apps”) such as, but not limited to, a web browser,messaging application, electronic mail (email) application, ane-commerce site application (also referred to as a marketplaceapplication), and the like. In some embodiments, if the e-commerce siteapplication is included in a given one of the device machines 110, 112,then this application is configured to locally provide the userinterface and at least some of the functionalities with the applicationconfigured to communicate with the networked system 102, on an as neededbasis, for data and/or processing capabilities not locally available(such as access to a database of items available for sale, toauthenticate a user, to verify a method of payment, etc.). Conversely ifthe e-commerce site application is not included in a given one of thedevice machines 110, 112, the given one of the device machines 110, 112may use its web browser to access the e-commerce site (or a variantthereof) hosted on the networked system 102. Although two devicemachines 110, 112 are shown in FIG. 1, more or less than two devicemachines can be included in the system 100.

An Application Program Interface (API) server 114 and a web server 116are coupled to, and provide programmatic and web interfaces respectivelyto, one or more application servers 118. The application servers 118host one or more marketplace applications 120 and payment applications122. The application servers 118 are, in turn, shown to be coupled toone or more databases servers 124 that facilitate access to one or moredatabases 126.

The marketplace applications 120 may provide a number of e-commercefunctions and services to users that access networked system 102.E-commerce functions/services may include a number of publisherfunctions and services (e.g., search, listing, content viewing, payment,etc.). For example, the marketplace applications 120 may provide anumber of services and functions to users for listing goods and/orservices or offers for goods and/or services for sale, searching forgoods and services, facilitating transactions, and reviewing andproviding feedback about transactions and associated users.Additionally, the marketplace applications 120 may track and store dataand metadata relating to listings, transactions, and user interactions.In some embodiments, the marketplace applications 120 may publish orotherwise provide access to content items stored in application servers118 or databases 126 accessible to the application servers 118 and/orthe database servers 124. The payment applications 122 may likewiseprovide a number of payment services and functions to users. The paymentapplications 122 may allow users to accumulate value (e.g., in acommercial currency, such as the U.S. dollar, or a proprietary currency,such as “points”) in accounts, and then later to redeem the accumulatedvalue for products or items (e.g., goods or services) that are madeavailable via the marketplace applications 120. While the marketplaceand payment applications 120 and 122 are shown in FIG. 1 to both formpart of the networked system 102, it will be appreciated that, inalternative embodiments, the payment applications 122 may form part of apayment service that is separate and distinct from the networked system102. In other embodiments, the payment applications 122 may be omittedfrom the system 100. In some embodiments, at least a portion of themarketplace applications 120 may be provided on the device machines 110and/or 112.

Further, while the system 100 shown in FIG. 1 employs a client-serverarchitecture, embodiments of the present disclosure is not limited tosuch an architecture, and may equally well find application in, forexample, a distributed or peer-to-peer architecture system. The variousmarketplace and payment applications 120 and 122 may also be implementedas standalone software programs, which do not necessarily havenetworking capabilities.

The web client 106 accesses the various marketplace and paymentapplications 120 and 122 via the web interface supported by the webserver 116. Similarly, the programmatic client 108 accesses the variousservices and functions provided by the marketplace and paymentapplications 120 and 122 via the programmatic interface provided by theAPT server 114. The programmatic client 108 may, for example, be aseller application (e.g., the TurboLister application developed by eBayInc., of San Jose, Calif.) to enable sellers to author and managelistings on the networked system 102 in an off-line manner, and toperform batch-mode communications between the programmatic client 108and the networked system 102.

FIG. 1 also illustrates a third party application 128, executing on athird party server machine 130, as having programmatic access to thenetworked system 102 via the programmatic interface provided by the APIserver 114. For example, the third party application 128 may, utilizinginformation retrieved from the networked system 102, support one or morefeatures or functions on a website hosted by the third party. The thirdparty website may, for example, provide one or more promotional,marketplace, or payment functions that are supported by the relevantapplications of the networked system 102.

FIG. 2 illustrates a block diagram showing components provided withinthe networked system 102 according to some embodiments. The networkedsystem 102 may be hosted on dedicated or shared server machines (notshown) that are communicatively coupled to enable communications betweenserver machines. The components themselves are communicatively coupled(e.g., via appropriate interfaces) to each other and to various datasources, so as to allow information to be passed between theapplications or so as to allow the applications to share and accesscommon data. Furthermore, the components may access one or moredatabases 126 via the data servers 128.

The networked system 102 may provide a number of publishing, listing,and/or price-setting mechanisms whereby a seller (also referred to as afirst user) may list (or publish information concerning) goods orservices for sale or barter, a buyer (also referred to as a second user)can express interest in or indicate a desire to purchase or barter suchgoods or services, and a transaction (such as a trade) may be completedpertaining to the goods or services. To this end, the networked system102 may comprise at least one publication engine 202 and one or moreselling engines 204. The publication engine 202 may publish information,such as item listings or product description pages, on the networkedsystem 102. In some embodiments, the selling engines 204 may compriseone or more fixed-price engines that support fixed-price listing andprice setting mechanisms and one or more auction engines that supportauction-format listing and price setting mechanisms (e.g., English,Dutch, Chinese, Double, Reverse auctions, etc.). The various auctionengines may also provide a number of features in support of theseauction-format listings, such as a reserve price feature whereby aseller may specify a reserve price in connection with a listing and aproxy-bidding feature where by a bidder may invoke automated proxybidding. The selling engines 204 may further comprise one or more dealengines that support merchant-generated offers for products andservices.

A listing engine 206 allows sellers to conveniently author listings ofitems or authors to author publications. In one embodiment, the listingspertain to goods or services that a user (e.g., a seller) wishes totransact via the networked system 102. In some embodiments, the listingsmay be an offer, deal, coupon, or discount for the good or service. Eachgood or service is associated with a particular category. The listingengine 206 may receive listing data such as title, description, andaspect name/value pairs. Furthermore, each listing for a good or servicemay be assigned an item identifier. In other embodiments, a user maycreate a listing that is an advertisement or other form of informationpublication. The listing information may then be stored to one or morestorage devices coupled to the networked system 102 (e.g., databases126). Listings also may comprise product description pages that displaya product and information (e.g., product title, specifications, andreviews) associated with the product. In some embodiments, the productdescription page may include an aggregation of item listings thatcorrespond to the product described on the product description page.

The listing engine 206 also may allow buyers to conveniently authorlistings or requests for items desired to be purchased. In someembodiments, the listings may pertain to goods or services that a user(e.g., a buyer) wishes to transact via the networked system 102. Eachgood or service is associated with a particular category. The listingengine 206 may receive as much or as little listing data, such as title,description, and aspect name/value pairs, that the buyer is aware ofabout the requested item. In some embodiments, the listing engine 206may parse the buyer's submitted item information and may completeincomplete portions of the listing. For example, if the buyer provides abrief description of a requested item, the listing engine 206 may parsethe description, extract key terms and use those terms to make adetermination of the identity of the item. Using the determined itemidentity, the listing engine 206 may retrieve additional item detailsfor inclusion in the buyer item request. In some embodiments, thelisting engine 206 may assign an item identifier to each listing for agood or service.

In some embodiments, the listing engine 206 allows sellers to generateoffers for discounts on products or services. The listing engine 206 mayreceive listing data, such as the product or service being offered, aprice and/or discount for the product or service, a time period forwhich the offer is valid, and so forth. In some embodiments, the listingengine 206 permits sellers to generate offers from the sellers' mobiledevices. The generated offers may be uploaded to the networked system102 for storage and tracking.

Searching the networked system 102 is facilitated by a searching engine208. For example, the searching engine 208 enables keyword queries oflistings published via the networked system 102. In example embodiments,the searching engine 208 receives the keyword queries from a device of auser and conducts a review of the storage device storing the listinginformation. The review will enable compilation of a result set oflistings that may be sorted and returned to the client device (e.g.,device machine 110, 112) of the user. The searching engine 308 mayrecord the query (e.g., keywords) and any subsequent user actions andbehaviors (e.g., navigations).

The searching engine 208 also may perform a search based on the locationof the user. A user may access the searching engine 208 via a mobiledevice and generate a search query. Using the search query and theuser's location, the searching engine 208 may return relevant searchresults for products, services, offers, auctions, and so forth to theuser. The searching engine 208 may identify relevant search results bothin a list form and graphically on a map. Selection of a graphicalindicator on the map may provide additional details regarding theselected search result. In some embodiments, the user may specify aspart of the search query a radius or distance from the user's currentlocation to limit search results.

The searching engine 208 also may perform a search based on an image.The image may be taken from a camera or imaging component of a clientdevice or may be accessed from storage.

In a further example, a navigation engine 210 allows users to navigatethrough various categories, catalogs, or inventory data structuresaccording to which listings may be classified within the networkedsystem 102. For example, the navigation engine 210 allows a user tosuccessively navigate down a category tree comprising a hierarchy ofcategories (e.g., the category tree structure) until a particular set oflisting is reached. Various other navigation applications within thenavigation engine 210 may be provided to supplement the searching andbrowsing applications. The navigation engine 210 may record the varioususer actions (e.g., clicks) performed by the user in order to navigatedown the category tree.

Additional modules and engines associated with the networked system 102are described below in further detail, it should be appreciated thatmodules or engines may embody various aspects of the details describedbelow.

FIG. 3 illustrates a block diagram showing image processing and imagedata usage functionalities/operations implemented in modules andlibraries/data structures/databases according to some embodiments. Themodules comprise one or more software components, programs,applications, apps, or other units of code base or instructionsconfigured to be executed by one or more processors included in theapplication servers 118, device machine 110, and/or device machine 112.The modules include an image capture module 302, a swatch extractionmodule 304, a color histogram module 306, a pattern module 308, anindexing module 310, an orientation histogram module 312, a dominantcolor module 314, a comparison module 316, a re-rank module 318, acomplement module 320, a browser plugin module 322, and a user interestdetection module 324. The modules 302-324 can communicate with each ofan inventory database 300 and image index database 301, in whichdatabases 300, 301 may be included in the databases 126. Althoughmodules 302-324 are shown as distinct modules in FIG. 3, it should beunderstood that modules 302-324 may be implemented as fewer or moremodules than illustrated. It should also be understood that any ofmodules 302-324 may communicates with one or more components included inthe system 100, such as database servers 124, application servers 118,third party server 130, device machine 110, or device machine 112.Similarly, databases 300, 301 are shown as distinct databases in FIG. 3.However, it is understood that the content of databases 300, 301 may bestored in fewer or more databases than illustrated.

In some embodiments, one or more of modules 302-324 are downloaded froman e-commerce site appropriate for the type of computing device. Forexample, if the device machine 110 (or device machine 112) comprises aniOS-type device (e.g., iPhone or the iPad), then the modules (which canbe packaged as part of an e-commerce site app) can be downloaded fromiTunes. Similarly, if the device machine 110 (or device machine 112)comprises an Android-type device, then the modules can be downloadedfrom the Android Marketplace. The device machine 110 and/or devicemachine 112 has communication capabilities with servers or databases ata remote location (e.g., databases 126, database servers 124, API server114, web server 116) to access data and/or processing capabilities tofacilitate image capture, image processing, and use of image data fromimage processing as described in detail below.

In other embodiments, one or more of modules 302-324 may be hosted onthe application servers 118 and no download of the modules is requiredon the device machines 110, 112. Instead, the modules may be accessed bydevice machines 110, 112 using a web browser over the network 104. Instill other embodiments, some of the modules may be included in thedevice machines 110, 112 while other of the modules may be included inthe application servers 118; the device machines 110, 112 communicatingwith the application servers 118 to together provide the appropriatefunctionalities.

FIG. 4 illustrates an example flow diagram 400 for image processing anddetermination of image matches implemented by the modules of FIG. 3according to some embodiments. The operations of the flow diagram 400may be performed by the device machine 110, device machine 112, and/or aserver included in the networked system 102 (e.g., API server 114, webserver 116, or application servers 118). Operations/functionalities offlow diagram 400 can be classified into two phases: an indexing phase440 and a matching phase 442. The indexing phase 440 comprises offlineimage processing of inventory images (e.g., images corresponding toitems offered for sale stored in the inventory database 300) by, forexample, application servers 118 in order to obtain image feature data.The obtained image feature data can be indexed into the image indexdatabase 301 and subsequently used for faster look-up during thematching phase. In some embodiments the inventory images can beprocessed in one or more batches. All images of certain categories ofinventory items (e.g., clothing, textiles, bedding, shoes, bags, etc.)are indexed based on their features/attributes such as, but not limitedto, color distribution, dominant color, orientation histogram, pattern,item category, and the like. Once the indexing phase 440 is complete,the matching phase 442 can be triggered by receipt of a query comprisingan image (referred to as a query image). The image index database 301 isaccessed to find the closest matching inventory image(s). Theseinventory image(s) are presented as matching results to the query image.Blocks 402 a-420 a relates to the indexing phase 440, and blocks 402b-430 b relates to the matching phase 442. The indexing phase 440operations are first described below followed by the matching phase 442operations.

For the indexing phase 440, at a block 402 a, the networked system 102(e.g., application servers 118) retrieves an inventory image from theinventory database 300. The inventory database 300 contains informationabout each item offered for sale (or listed) by the e-commerce ormarketplace system 100. Information about each item offered for saleincludes, but is not limited to, one or more item images, itemdescription, price, brand and model name, seller information, and thelike. The inventory image corresponds to an item from any of, but notlimited to, the following item categories: clothing, textiles, bedding,shoes, bags, upholstery, electronics, home and garden, or collectibles.Although operations taken on a single inventory image is discussedherein, it is understood that a plurality of inventory images can beprocessed simultaneously with each other in batch jobs. Operations takenon a given single inventory image is for ease of discussion only.

At a block 404 a, the swatch extraction module 304 is configured toidentify and perform swatch extraction on the retrieved inventory image.It is not uncommon for inventory images to have poor lighting, lowcontrast, and/or cluttered background. Such less than ideal photographsof inventory items make it difficult to capture feature information. Inorder to compensate for such shortcomings, the central part of theinventory image is used to perform feature extraction. The central partof the inventory image is more likely to contain the inventory item thanother parts of the inventory image. The background is also automaticallyremoved. A sampling mask (e.g., a rectangular region) located at thecentral part of the inventory image is used to identify a swatch orportion of the inventory image to use for subsequent feature extractionoperations. The sampling mask provides the spatial sampling of theinventory image.

For example, inventory images of clothing can be accurately sampled fromwithin the sampling mask area. FIG. 5A illustrates sample mean of redgreen blue (RGB) images 502 as well as that of sample means of masks 505after background removal 504 for each of a plurality of dress images.The sample mean of masks 505 is an estimate of the spatial prior foreach dress style 506 (dress styles based length of the sleeves none,short, long, etc.). Thus, the change of sampling from the clothing ishigh if the sample is taken from the central region of the image. Thecenter-sampling strategy works well even in the presence of a complexcluttered background. FIG. 5B illustrates a clothing image that isspatially sampled appropriately (with sampling mask 505 located in thecentral portion of the image) even though the image has a clutteredbackground. FIG. 5C illustrates a clothing image containing multipleobject instances (e.g., multiple views of the article of clothing withina single image). When the background is simple, it can be removed andthe swatch can still be sampled from within the sampling mask 505region.

Once the sampling area of the inventory image has been determined andextracted—the area within the sampling mask—such sampling area is usedfor various image feature extraction and identification. At a block 506a, the color histogram module 306 is configured to extract the colorhistogram of the inventory image (using the information contained in thesampling mask area). Extraction of color histogram comprises theextraction or identification of the color distribution of the itemfeatured in the inventory image.

The choice of color representation is relevant for extraction of colordistribution. Color can be represented in various ways. A common colorspace is the RGB color space, where all colors are represented withrespect to three color channels red, green, and blue in 3-dimensions.With RGB color space, however, illumination or lighting variations in animage can affect the three color channels even though a color may beperceived by a user as a single/same color. More than one shade of thecolor may appear to exist due to lighting variations such as shadows orattenuation of strength of illumination over space. FIG. 5D illustratesa sampling mask 510 over a clothing item image. The color histogram ordistribution corresponding to the content of the sampling mask 510 inRGB color space is shown in plot 512. Because the actual color withinthe sampling mask 510 is a single color (the clothing item is notmulti-colored), plot 512 should show a single peak at a single maximumvalue. Instead, plot 512 shows a number of peaks over a wide range ofmaximum values due to lighting variations interpreted in RGB color spaceas different shades of a color or different colors.

Other color spaces include hue, saturation, and value (HSV); hue,saturation, and lightness/luminance (HSL); hue, saturation, andintensity (HSI); lightness, color-opponent dimension, and color-opponentdimension (Lab); color space adopted by the International Commission onIllumination (LUV); and luminance, blue-difference chroma, andred-difference chroma (YC_(b)C_(r)). Lab, LUV, and YC_(b)C_(r) areexamples of non-linear color spaces. Each of these color spaces map RGBvalues to 2-dimensional chrominance space and a singleluminance/brightness channel captures most of the lighting variations.FIG. 5E illustrates the 2-dimensional chrominance spaces for HSV, LUV,YC_(b)C_(r), and Lab color spaces in respective columns. Each row showsthe respective color spaces at different brightness levels, thebrightness level increasing in the direction of the bottom of the page.The chrominance space dims with reduction in brightness of HSV, LUV, andLab, but not for YC_(b)C_(r). FIG. 5F illustrates an alternative view ofHSV color space. Saturation is represented by 5, the radius (in ∞-norm)of Red, Green, Blue values relative to a value V. Hue represented by Hvaries angularly. For a given V, the area of annular disc increases as Sincreases.

In one embodiment, the inventory image (more particularly, within thesampling mask area of the inventory image) is sampled or scanneduniformly (also referred to as uniform sampling) using HSV color space.The pixels from the image are sampled. This is defined by the samplingmask, which defines an estimate of which pixels of the image are ofinterest. Once these pixels are sampled, information (e.g., imagefeatures or attributes) is extracted from each of these pixels. Theinformation can be extracted based on the pixel or the pixel along withits immediate neighbors. Information about color is extracted on a pixelby pixel basis. And then combined, consolidated, or pooled into acollection of values (also referred to as features). In the case ofcolor features, histogram techniques are used to obtain, for example, acolor histogram or an orientation histogram. A histogram comprises aconsolidation of occurrences of various values an item, such as a pixel,can take. For example in the case of a color hue, it can have a valuebetween 0 and 1, with an infinite number of possibilities of valuespossible between 0 and 1. One approach is to quantize values between 0and 1 to a finite set of possible values—such as 0, 0.1, 0.2, 0.3, 0.4,0.5, 0.6, 0.7, 0.8, 0.9, and 1. In one embodiment, the hue dimension isquantized into 24 values, saturation dimension is quantized into 8values, and value dimension is quantized into 8 values. Suchquantization may be referred to as the second sampling, the firstsampling comprising a sampling of certain pixels of the image and thesecond sampling comprising a sampling of the value of a pixel quantity.Fewer samples are obtained for luminance/brightness because it is theleast reliable of the three dimensions for HSV color space, owing tovariations introduced by potential lighting variations.

In FIG. 5E, the three channels of HSV color space are represented by ahorizontal axis 520 for hue, a vertical axis 522 for saturation, and athird axis 524 for value or brightness. Hue highly correlates to colorfrequency, similar to the colors of a rainbow. Saturation represents thepurity of a color for a given hue. For example, a low-saturation red isless red than high-saturation red; however, both are still red, onlydiffering in their vividness or colorfulness. Colors tend to beconfusing for low-saturation colors in the HSV space. This region can behandled separately.

The mathematical formulas for hue H, saturation S, and value V channelsof HSV color space are shown below:

$\begin{matrix}{V = {\max \left( {R,G,B} \right)}} \\{S = \left\{ \begin{matrix}{\frac{V - {\min \left( {R,G,B} \right)}}{V},} & {{{if}\mspace{14mu} v} > 0} \\{0,} & {otherwise}\end{matrix} \right.} \\{H = \left\{ \begin{matrix}{\frac{60\left( {G - B} \right)}{S},} & {{{if}\mspace{14mu} V} = R} \\{{120 + \frac{60\left( {B - R} \right)}{S}},} & {{{if}\mspace{14mu} V} = G} \\{{240 + \frac{60\left( {R - G} \right)}{S}},} & {{{if}\mspace{14mu} V} = B}\end{matrix} \right.}\end{matrix}$

where R, G, and B are the RGB values in RGB color space. When H<0,assign H←H+360. Thus, 0≤V≤1, 0≤S≤1 and 0≤H≤360.

Based on the formulas above, it can be seen that for small values of H,all three values R, G, and B are similar. The same is also true forsmall values of S, since the maximum and minimum values of R, G, and Bwill be similar. This means that, due to noise, each of R, G, and Bvalues may be dominant spuriously. Thus, hue has discontinuous values(see the conditional assignment for hue in the formulas above). In otherwords, hue is not reliable when saturation is low. When saturation islow, color will lack vividness and look grayish. This gray region can behandled separately from the colorful region.

Because sampling in full three dimension space as cross-product createsa sparse color histogram, the color histogram module 306 samples theinventory image at least three separate times, once for each of the hue,saturation, and value channel/dimension of the HSV color space. Then theone-dimensional (1D) histograms of hue, saturation, and value arestacked or overlaid over each other. The result is a relatively denseand accurate color histogram of the inventory image in all threedimensions of the HSV color space. Moreover, because the total number ofbins in this stacked 1D sampling scheme is much smaller than the full 3Dsampling scheme (n_(H)+n_(S)+n_(V) for the 1D sampling scheme vs.n_(H)*n_(S)*n_(V) for the 3D sampling scheme), the memory and processingrequirements are significant smaller. This aids in batch indexing alarge number of inventory images at the same time.

For example, assume there are five possible values for each space of a3D color space: {0, 0.25, 0.5, 0.75, 1}. In the case of HSV color space,each of H, S, and V can take values in {0, 0.25, 0.5, 0.75, 1}. Thenumber of possible combinations (which is a cross product) is 5³=125.Very few such entries occur frequently. Hence, it is sparse, e.g., lotsof entries have 0 accumulations of number of occurrence. This can bewaste of space (in bytes). Thus, treat each dimension separately, to geta much compact representation. With this technique, only 5+5+5=15 valuesare required.

FIG. 6 illustrates details of block 406 a, b according to someembodiments. In one embodiment, at a sub-block 450, the 1D samplingscheme for HSV space uses uniformly spaced bins along the hue,saturation, and value axes, respectively. As shown in FIG. 5G, the 1Dsampling scheme for a given inventory image is implemented using anumber of bins along the hue axis of n_(H)=24, a number of bins alongthe saturation axis of n_(S)=8, and a number of bins along the valueaxis of n_(V)=8. This is for “colorful” or color pixels, those withsaturation of at least 6%. For pixels with saturation less than 6%, theyare deemed to be “gray” and are binned separately into 8 bins. Thus, atotal of 48 bins=24+8+8+8 are used to extract the provisional colorhistogram of the inventory image.

Once the bins are accumulated, at a sub-block 452, weights are appliedas follows: 0.4 for H values for colorful pixels, 0.2 for S values forcolorful pixels, 0.1 for V values for colorful pixels, and 0.3 for Vvalues for gray pixels. Note that the obtained values within each groupof bins are not normalized so that the ratio of color pixels to graypixels is retained and encoded in the stacked histogram.

Then at a sub-block 454, the 1D samples that have been weight adjustedare stacked or combined together to generate a resulting stacked 1Dhistogram corresponding to the three dimensions/channels of the HSVcolor space. FIG. 5H illustrates an example stacked 1D histogram 532corresponding to an inventory image 530 of a multi-colored dress. Thehorizontal axis represents the bins (a total of 48 bins) for H, S, and Vfor color pixels (bin group 533 for H, bin group 534 for S, bin group535 for V) and also for V for gray pixels (bin group 536). The binscorrespond to different colors in the HSV color space. The vertical axisrepresents the amount of spatial area of a given color relative to othercolors sampled on the inventory image 530. Thus, high peaks representcolors that are more prominently present than other colors for themulti-colored dress. As expected, there are no gray colors in themulti-colored dress, and the histogram 532 reflects the absence of gray(e.g., zero or very low amplitudes in bin group 536). The stacked 1Dhistogram presents information about the image in a relatively compactform, instead of storing values of all pixels of the sampled portion ofthe image. The vertical axis represents the number of pixels that takethe corresponding value on the horizontal axis, with the exception thatgroups 533, 534, 535, and 536 are scaled/weighted differently by 0.4,0.2, 0.1 and 0.3, respectively.

Hence, the resulting stacked 1D histogram not only identifies the colors(as well as degree of saturation, brightness, and/or other dimensions ofthe particular color space that the image is sampled for) present withinthe sampling mask of the inventory image (which is representative of theentire item being featured in the inventory image) but also the amountof spatial area each of the present colors occupy relative to eachother.

In alternative embodiments, the inventory image can be uniformly sampledusing a color space other than HSV color space. In still otherembodiments, the inventory image can be non-uniformly sampled using LUVcolor space or other color space appropriate for non-uniform sampling.

Next at a block 408 a, the pattern module 308 is configured to detectthe presence of and type of pattern in the inventory image. Patternsincluding, but not limited to, polka dots, stripes, check, plaid,floral, and other repeating patterns can be detected. In one embodiment,an edge map of the inventory image is generated using a Canny edgedetector to obtain a Canny edge map corresponding to the inventory image(sub-block 462 of block 408 a in FIG. 7). The Canny edge map, in turn,is used to detect a polka dot pattern in the inventory image. FIG. 5Iillustrates an image 540 showing a polka dot top. A Canny edge map 542corresponding to the image 540 comprises a faithful line drawing of theedges included in the image with the colors removed.

At a sub-block 464 in FIG. 7, the pattern module 308 analyzes the shapesin the Canny edge map 542 to classify them as circles or not circles. Inparticular, the pattern module 308 studies the properties of blobs 543in the Canny edge map 542 to determine whether they are circles. Onlyblobs 543 that are large enough but not too large are retained foranalysis. Blobs 543 that are too cluttered are eliminated for analysis.Circularity of each of the blobs 543 retained for analysis iscalculated. Circularity is used as a measure of closeness of circularshape. Circularity has a value between 0 and 1, with 1 corresponding toa perfect circle. For a given perimeter of a shape, a circle has themaximum area among all the shapes. The circularity of each of the blobs543 of interest is calculated as a ratio of the area of the given blob543 to the area of a circle having the same perimeter as the given blob543. A circularity value of 0.8 or higher is considered to besufficiently circular to be deemed a circle. This takes into accountcases where a circle appears to be an ellipse (or other distorted shape)due to folds in the clothing, position of the model wearing theclothing, or other transformation artifacts. The radii of the blobs 543determined to be circles are also estimated to remove outliers anddetermine the repeatability of spacing between adjacent “circles.”

Based on such analysis (e.g., identification of sufficient numbers ofcircles and set spacing between circles), the pattern module 308determines whether a polka dot pattern is present in the inventory image(sub-block 466). FIG. 5I shows an image 544 in which detected polka dots546 is shown.

If a pattern (e.g., polka dot pattern) is detected (yes branch of block408 a), the pattern module 308 determines a confidence score or levelfor the polka dot pattern detection in the inventory image (block 410a). As an example, if the blobs 543 have an average circularity value of0.9, the corresponding confidence score may be higher than blobs ofanother image having an average circularity value of 0.82. At the block410 a, the indexing module 310 is also configured to appropriately indexand add the image feature data corresponding to the inventory image tothe image index database 301. Image feature data includes the colorhistogram (extracted in block 406 a), the pattern (detected in block 408a), and pattern confidence score. The inventory image can be indexedbased on one or more attributes. For example, the image can be indexedaccording to its clothing style (e.g., dress, top, pants, outerwear,bag, etc.), colors, pattern, and the like to facilitate rapid look-up ofmatching items.

Otherwise a (polka dot) pattern is not detected (no branch of block 408a), and the flow diagram 400 proceeds to a block 412 a. At the block 412a, the orientation histogram module 312 is configured to determine ormeasure the directionality of (other) pattern or texture as well as thecomplexity of the pattern/texture present in the inventory image.Details of the block 412 a are provided in FIG. 8. At a sub-block 470 ofFIG. 8, the orientation histogram module 312 is configured generate anedge map of the inventory image. As an example, the edge map may beobtained using a Canny edge detector. Next a sub-block 472, theorientation histogram module 312 estimates the x- and y-derivatives ofthe edge map using the Scharr operator. The x- and y-derivatives arethen used to determine the gradient and orientation of the edge map at asub-block 474.

At a sub-block 476, the orientation histogram module 312 applies aweight to each edge pixel (of the edge map) that is its edge magnitude.The result is a weighted orientation histogram. Such weighing of theedge pixels favors (or magnifies the existence of) strong edges andreduces the influence of weak edges, such as those caused by ruffles orshadows. The weighted orientation histogram is also normalized by thenumber of edge pixels.

Then at a sub-block 478, the orientation histogram module 312 sums thebins in the weighted orientation histogram to obtain a measure of theamount of busy pattern or texture. The greater the summed value, thegreater the measure of busy pattern/texture. If a pattern/texture has astrong directionality, such as for a striped shirt, the entropy of theorientation histogram will be low, and vice versa. The entropy of theorientation histogram can be used as a measure of confidence in theorientation histogram.

FIG. 5J illustrates an example implementation of block 412 a accordingto some embodiments. An image 550 shows a horizontally striped shirt.After application of a Canny map detector (sub-block 470), a Canny edgemap 552 corresponding to the image 550 is generated. X- andy-derivatives of the Canny edge map 552 are shown as images 554 and 556.Notice that the y-derivative image 556 shows strong directionality inthe horizontal direction consistent with the item being a horizontallystriped shirt, while the x-derivative image 554 shows little feature ofinterest. Horizontal stripes have low x-derivative and high y-derivativefeatures. This is also evident from the concentration of peaks and highpeaks being in the center pat of the orientation histogram 558 ofgradient orientation. Because the magnitude of the edge is high forhorizontal stripes, applying such edge magnitude to the orientationhistogram 558 results in a weighted orientation histogram 559 thatmagnifies those peaks and further reduces the lows.

Once the orientation histogram is extracted, a check is performed todetermine whether the inventory image has low or high spatial frequencyat a block 414 a. Low spatial frequency corresponds to a pattern/texturethat is minimally busy or in which the sum of the bins of the weightedorientation histogram is a low value. The weighted orientation histogramtakes into account the strength of the discontinuities. Thus, for lowcontrast, but still a high spatial frequency pattern, this histogramshows a low value. This is a tradeoff in order to suppress noise fromshadows, folds, and wrinkles. Examples of low spatial frequency includeitems with no pattern/texture or solid color items. Conversely, highspatial frequency (e.g., images with large amount of color spatialvariation) corresponds to a busy pattern/texture or in which the sum ofthe bins of the weighted orientation histogram is a high value. Examplesof high spatial frequency include items that have small and frequentpolka dots, stripes, geometric patterns, animal prints, floral patterns,and the like.

For low spatial frequency, the flow diagram 400 proceeds to block 416 a.At block 416 a, the dominant color module 314 is configured to extractthe dominant color of the inventory image. The dominant color comprisesthe color that is present on the most spatial area relative to othercolor(s) within the swatch area of the inventory image. Thus, even ifthe sampling mask area includes more than one color, one of those colorscan be the dominant color.

In one embodiment, the dominant color module 314 is configured tocluster the colors from all pixels in the swatch area in accordance withLUV color space, and identify the largest cluster from among suchclusters of colors. The color corresponding to the largest cluster isthe dominant color. Using LUV color space, the colors are clusteredK-means with k=8. The initial cluster centers are selected using theK-means++ scheme, where centroids are picked to be as far apart aspossible given the first randomly picked centroid. Details about theK-means++ scheme is provided in David Arthur, et at, “K-means++: Theadvantages of careful seeding,” SODA 2007 Proceedings of the 18^(th)annual ACM-SIAM symposium on discrete algorithms, 1027-1035 (2007).

FIG. 5K illustrates an example image 560 and its swatch area 561. Ascatter plot 562 corresponding to the swatch area 561 is shown, thescatter plot 562 shown in the U-V space of the LUV color space. Eachpoint on the scatter plot 562 corresponds to a pixel within the swatcharea 561, in which each point is color coded with the true color of thecorresponding pixel. As can be seen in scatter plot 562, thelargest/dominant cluster of a single color is cluster 563. The color ofcluster 563 is a coral color 564, and this is the dominant color of theswatch area 561 and by extension, of the image 560. The dominant coloridentified via the scatter plot 562 is the same as the color 564 thatoccupies the largest area within the swatch area 561.

In an alternative embodiment, the dominant color module 314 isconfigured to re-use the hue H values in the stacked 1D histogram fromblock 406 a. With reference to FIG. 5H, the bin within the bin group 533(corresponding to the H values for color pixels within the swatch area)having the highest peak represents the dominant color for the swatcharea/image 530. In FIG. 5H, the highest peak within bin group 533 is atbin 17. Bin 17 corresponds to a bluish purple color. And in fact, thedominant color is image 530 is a bluish purple color.

Once the dominant color is identified, at a block 418 a, the dominantcolor module 314 determines a confidence score or level for the dominantcolor detected in the inventory image using the color histogram. As anexample, confidence of dominant color is low if the image has multiplecolors with similar spatial span (as shown in the color histogram).Conversely the confidence is high if the image is a solid color.

The confidence score can be a binary value (low or high) or it can be aquantitative value between, for example, a score of 0 to 1. For block418 a, the indexing module 310 is also configured to appropriately indexand add the image feature data corresponding to the inventory image tothe image index database 301. Image feature data includes the colorhistogram, orientation histogram, dominant color, and dominant colorconfidence score. The inventory image can be indexed based on one ormore attributes. For example, the image can be indexed according to itsclothing style (e.g., dress, top, pants, outerwear, bag, etc.), colorhistogram, dominant color, and the like to facilitate rapid look-up ofmatching items.

If the spatial frequency is high, then the orientation histogram module312 is configured to determine a confidence score or level for theorientation detected in the inventory image (block 420 a). An image withhigh spatial frequency may or may not also have strong directionality ororientation. As an example, a horizontal striped shirt exhibits strongorientation, and thus its orientation confidence score is high. Theconfidence score can be a binary value (low or high) or it can be aquantitative value between, for example, a score of 0 to 1. For block420 a, the indexing module 310 is also configured to appropriately indexand add the image feature data corresponding to the inventory image tothe image index database 301. Image feature data includes the colorhistogram, orientation histogram, orientation information, andorientation confidence score. The inventory image can be indexed basedon one or more attributes. For example, the image can be indexedaccording to its clothing style (e.g., dress, top, pants, outerwear,bag, etc.), color histogram, orientation, high spatial frequency, andthe like to facilitate rapid look-up of matching items.

Thus, image features of a given inventory image are extracted/identifiedand such image features are stored in the image index database 301appropriated indexed to facilitate rapid look-up in response to queryimages. By repeating blocks 402 a-420 a as many times as needed, all ofthe inventory images corresponding to certain item types can besimilarly indexed.

With the inventory images indexed, FIG. 4 will be described in thecontext of the matching phase 442. At a block 402 b, the image capturemodule 302 is configured to receive a query image. In one embodiment,the query image is sent by a device machine 110 or 112 to the networkedsystem 102. A user of the device machine 110 or 112 (e.g., a smartphone)may take a photograph of a color, pattern, textile, clothing, a portionof a clothing such as a sleeve, or the like to capture an image ofinterest using the camera included in the device machine 110 or 112. Thephotograph is transmitted to the networked system 102, to be the queryimage, for image feature extraction and to return inventory imagesmatching the query image.

For blocks 404 b-408 b and 412 b-410 b, the same operations as describedabove for blocks 404 a-408 a and 412 a-416 a are performed except theoperations are taken on the query image instead of an inventory image.If a pattern is detected at block 408 a, then at block 410 b, thepattern module 308 determines the pattern confidence score of the queryimage similar to the determination described in block 410 a. For block410 b, the comparison module 316 is configured to compare the colorhistogram of the query image to the color histograms of the inventoryimages to find one or more inventory images matching the query image.Possible similarity schemes include, but are not limited to, cosinesimilarity (inner product) correlation, Chi-squared correlation,intersection, and Bhattacharyya distance.

In one embodiment, the similarity score is calculated to be thecomplement of the Bhattacharyya distance. The Bhattacharyya distancecomprises the square root of the complement of cosine similarity of thesquare root of the color histograms. The similarity score S betweencolor histograms H₁ and H₂ is expressed as follows:

${S\left( {H_{1},H_{2}} \right)} = {1 - \sqrt{1 - {\frac{1}{\sqrt{{{\overset{\_}{H}}_{1} \cdot {\overset{\_}{H}}_{2}}N^{2}}}{\sum\limits_{I}\sqrt{{H_{1}(I)} \cdot {H_{2}(I)}}}}}}$

where N is the number of bins.

The similarity score (e.g., between 0 for completely dissimilar and 1for exactly the same) is assigned for each pair of the query image andan indexed inventory image. The similarity scores are sorted so thatmatching results can be displayed on the device machine 110/112 in theorder of similarity to the input query image.

Then at a block 422 b, the comparison module 316 provides the matchingresults to the device machine 110/112 for display on the screen of thedevice machine 110/112. The matching results (e.g., inventory images andpossibly associated item information) are displayed in order of degreeof similarity to the query image. In some embodiments, the system or theuser may set an upper limit on the number of matching results that maybe presented in response to the query image.

If the query image has low spatial frequency (low branch of block 414b), then the query image's dominant color is extracted in block 416 b.Next at a block 418 b, the dominant color module 314 determines thedominant color confidence score using the color histogram from block 406b. If the confidence of the dominant color is low, the comparison module316 compares the color histogram of the query image to the colorhistograms of the indexed inventory images (block 428 b). Note that ifthe dominant color confidence score for the query image is aquantitative value, a pre-defined threshold value may be used todetermine if the score is above or below the threshold value todetermine a “high” or a “low,” respectively. A similarity score isgenerated for each pair of images as described above with respect toblock 410 b. The similarity scores are similarly ordered from highest tolowest scores. Then the inventory images with the highest similarityscores, the matching results, are provided to the device machine 110/112for display (block 422 b).

On the other hand if the confidence of the dominant color is high, thecomparison module 316 compares the dominant color of the query image tothe dominant color of each of the indexed inventory images (block 430b). A similarity score is assigned for each pair of the query image andindexed inventory image, and the similarity scores are ordered. Then theflow diagram 400 proceeds to block 422 b to provide the matching resultsto the device machine 110/112. Note that if an inventory image has lowconfidence for dominant color, it will be “penalized” or considered tobe dissimilar to a query image that has high confidence for dominantcolor.

At a block 420 b, the orientation histogram module 312 is configured todetermine the confidence score for the orientation of the query image,similarly as described above with respect to block 420 a. If theconfidence score for orientation/directionality is low, then at a block424 b, the comparison module 316 compares the color histogram of thequery image to the color histograms of the indexed inventory images.Note that if the confidence score for the query image is a quantitativevalue, a pre-defined threshold value may be used to determine if thescore is above or below the threshold value to determine a “high” or a“low,” respectively. A similarity score is generated for each pair ofimages as described above with respect to block 410 b. The similarityscores are similarly ordered from highest to lowest scores. Then theinventory images with the highest similarity scores, the matchingresults, are provided to the device machine 110/112 for display (block422 b).

On the other hand, if the confidence score fororientation/directionality is high, then at a block 426 b, thecomparison module 316 performs two comparisons for each pair of queryimage and an indexed inventory image: a comparison of the colorhistograms and also a comparison of the orientation histograms. Asimilarity score can be assigned for each comparison of the (color ororientation) histograms using the similarity score formula discussedabove with respect to block 410 b. For each image pair, the finalsimilarity score is the sum of the color histogram comparison similarityscore and the orientation histogram comparison similarity score. Thenthe matching results are provided to the device machine 110/112 fordisplay at the block 422 b.

Accordingly, the image features (e.g., dominant color, colordistribution, pattern, orientation, etc.) of the query image dictatewhat image features of the inventory images to compare against to obtainsimilarity scores between pairs of query image and an inventory image.FIG. 9A illustrates a high-level block diagram of the matching phase, inwhich the device machine 110/112 interfaces with the networked system102 via a website 902 using a web browser. A query image 904 is sent tothe networked system 102 to extract features from the query image. Thenetworked system 102 uses the query image's extracted features to findmatches to inventory images. The top matches are returned to the devicemachine 110/112 formatted in a match results web page.

FIG. 9B illustrates various user interface (UI) screens displayed on thedevice machine 110/112 pertaining to the matching phase according tosome embodiments. In FIG. 9B, the device machine 110/112 has installedan app 905 to interface with the networked system 102 rather than accessthe website 902 hosted by the networked system 102. When a user launchesthe app 905 at the device machine 110/112 (as shown on screen 906), theapp 905 facilitates the user to input or otherwise specify a query image904. As an example, the app 905 may include camera capabilities (orotherwise access a separate camera app) to permit the user to take aphoto/image of something that is colored and/or patterned. The queryimage 904 shown on screen 907 comprises blue and cream horizontalstripes, likely from an article of clothing. The app 905 transmits thequery image 904 to the networked system 102, and in response, thenetworked system 102 performs image feature extraction operations andprovides matching results as discussed above.

Screen 908 shows matching results displayed on the device machine110/112. All items that are deemed to match the query image 904 matchesat least the color distribution of the query image 904. Continuing theabove example, each of the matching items contain at least blue andcream colors like the query image 904. The matching results may beorganized by item categories, such as “tops and blouses,” “dresses,”“coats and jackets,” “shirts,” “sweaters,” “T-shirts,” and the like. If,for example, the user selects the “tops and blouses” category on screen908, the matching items from that category are displayed to the user ata screen 910. The matching items are ordered from highest to lowestsimilarity score for that category. For each matching item, informationsuch as, but not limited to, a color image and one or more of an itemname, brand name, sizes, price, availability, or similarity score may bedisplayed. The user can select from among the displayed matching items,for example, a third ranked item 913 (a blue and white striped poloshirt), as shown in screen 912. In response, additional item details areprovided about the selected item at a screen 916. The user can navigatewithin the matching results to view one or more items of interest. Theuser can also purchase an item of interest within the app 905 (notshown). Thus, a user can simply take a photo of something having a colorand/or pattern that he/she likes, and the app 905 in conjunction withthe networked system 102 automatically returns items available forpurchase that matches that color and/or pattern.

When the inventory includes tens of thousands to millions of listeditems, the number of matching results for a given query image can beprohibitively high. Especially when the matching results are viewed onsmaller displays, as is common in smartphones and other mobile devices,viewing exhaustion can occur well before all of the matching items areviewed by the user. Even on larger displays, users are more likely toview top matching results than (much) lower matching results.Additionally in the context of apparel, when a user submits a queryimage containing a specific color and/or pattern, it is not obviouswhether the user is interested in just the color distribution, thepattern, the category of apparel, apparel in general, or a combinationof two or more features of the query image. In order to improve userengagement (which may increase the possibility of the user making apurchase) and personalize the user experience, user indications orpreferences during interaction with the provided matching results areused to re-rank or re-order the items within the initial matchingresults to better suit the user's interest.

FIG. 10 illustrates an example flow diagram 1000 for performingre-ranking operations according to some embodiments. During userinteraction with the matching results corresponding to a given queryimage at the device machine 110/112, the user may indicate a preferencefor or an interest in a particular item from among the match results.Continuing the example shown in FIG. 9B, the screen 914 includes itemdetails 915 corresponding to the selected third ranked item 913 (e.g.,larger image, additional images), a thumbnail of the query image 904,and a button 916 to affirmatively indicate user preference of that item.When the user actuates or clicks on button 916, the re-rank module 318at the networked system 102 receives the user indication/preference forthe particular item (block 1002). The user marked a particular item fromamong the match results as being relevant or of interest over otheritems in the match results.

Next at a block 10004, the re-rank module 318 saves the received userindication/preference. The screen 914 is correspondingly updated to ascreen 918. The button 916 in screen 914 is replaced with a re-rankbutton 917 in screen 918. Actuation of or clicking on the re-rank button917 by the user comprises a user request to re-rank or re-order thematch results based on the user preference for the particular item.Alternatively, re-rank button 917 may be omitted and initiation ofre-ranking may commence upon receiving user preference of a particularitem. Although not shown, a user preference for more than one itemwithin the match results can be indicated prior to initiation ofre-ranking.

if no re-rank request is received (no branch of block 1006), then thenetworked system 102 continues to provide matching results and beresponsive to user navigation within the matching results (block 1008).Otherwise a re-rank request is received (yes branch of 1006), and there-rank module 318 performs re-ranking of the (initial) matching resultsbased on the user preference(s) for one or more of particular itemswithin the matching results (block 1010). Then at a block 1012, there-rank module 318 provides the re-ranked matching results for displayon the device machine 110/112. The re-ranked matching results compriserefinement of the initial matching results in accordance with additionaluser input regarding item(s) of interest within the initial matchingresults.

Returning to FIG. 9B, screen 920 shows the initial matching resultsshown on screen 910 re-ordered based on the user preference for the(initially) third ranked item 913 (a blue and white striped polo shirt).Notice that the re-ranked results comprise the (initially) third rankeditem 913 as the first ranked item and more of the top results arestriped pattern items in comparison to the initial results.

As an example, the query image is a striped yellow dress and thenetworked system 102 weighs patterns at 60% and color at 40%.Accordingly, more striped clothing may show up as top results and notall of them yellow. The user viewing the recommendation results may pick(indicate user preference for) the one or more yellow dresses but notshow a preference for the non-yellow clothing items. The system re-ranksthe recommendation results based upon such preference, resulting inpushing up the listings of yellow dresses to the top over non-yellowclothing items.

The re-ranking operation can be repeated more than once for a givenmatching results in response to new or additional preference for certainitem(s) within the matching results provided by the user.

In some embodiments of block 1010, the re-rank module 318 may include asone or more of the top re-ranked results (e.g., within the top five, topten, or top twenty results) an item from among the initial matchingresults that differs from the image feature that the other top resultswill be selected for. An explore-exploit paradigm can be included in there-ranking. Instead of showing the most matching (and/or complementary)items, diverse items can also be included as top results to allow theuser to explore various diverse inventories pivoted to the query image.Continuing the example of FIG. 9B, the image feature of interest appearsto be the horizontal striped pattern of the third ranked item 913. Thus,the top re-ranked results shown on screen 920 are dominated byhorizontally striped items. However, rather than the horizontal stripepattern, the user may actually be expressing a preference for stripes ingeneral (horizontal stripes, vertical stripes, diagonal stripes, etc.)or items having approximately equal amounts of blue and cream colorsregardless of the pattern by indicating a preference for the thirdranked item 913. In order to test this “theory,” the re-ranked module318 can include an item from among the initial matching results that hasa blue and cream colored floral pattern or vertical stripe pattern asone of the top re-ranked results. Then if the user subsequentlyindicates a preference for such item, then the “theory” may be correctand the latest matching results can be further refined accordingly.

In addition to or as an alternative to providing matching results inresponse to receipt of a query image, the networked system 102 canprovide complementary results based on the query image. FIG. 11illustrates an example flow diagram 1100 relating to establishingrecommendation schemes in order to provide complementary resultsaccording to some embodiments. FIG. 11 is discussed below in conjunctionwith FIG. 4.

At a block 1102 of FIG. 11, the complement module 320 is configured toobtain purchasing behavior data (e.g., from purchases made using thenetworked system 102). Purchasing behavior data comprises correlationsbetween particular items that users have purchased together. Forexample, the purchasing behavior data may show that 60% of users thatpurchase a particular item A also purchase item B, either simultaneouslyor within a certain time period of each other. The purchasing behaviordata may also show that 10% of users that purchase item A also purchaseitem C, either simultaneously or within a certain time period of eachother. Thus, the correlation between items A and B is higher than thecorrelation between items A and C. Items A, B, and C may be samecategory of items to each other (e.g., dresses) or different categoriesof items to each other (e.g., a dress, shoes, and handbag).

Next at a block 1104, the complement module 320 determines rules (alsoreferred to as correlation rules, complementary rules, or recommendationrules) based on the purchasing behavior data. Continuing the aboveexample of items A, B, and C, because the image features of these itemshave been extracted and are indexed in the image index database 301, thecomplement module 320 can generate rules that comprise correlation rulesbetween image features corresponding to items A and B, for example, asopposed to between specific items A and B only. The rules define foreach of a given particular image index, which other image indices wouldbe best complements. An image index comprises a particular image featuredata profile as well as item category specification (e.g., dress, shirt,pants, dress, shoes, handbag, scarf, bedding, etc.). Using these rules,for example, recommendations can be made as to which handbags bestcomplement a particular skirt or which scarf is complementary to whichshoes, even if one or more of these exact items have no purchase historybecause the image feature data and item category data of the items areapplied against the rules to determine complements.

Once the purchasing behavior data-based rules are defined, thecomplement module 320 generates one or more template images (alsoreferred to as test images, complementary template images,recommendation template images, and similar variants) embodying therules established in block 1106. The template image(s) can be certaininventory image(s) from which a rule was derived. Or it could be a newimage that comprises a composite of a rule that generalized trends orother behavior extracted from the purchasing behavior data.Alternatively, instead of generating template image(s), template imageindices (including image feature data as would be obtained after animage is processed in the indexing phase 440 and item categoryinformation) embodying the rules can be generated instead.

The generated template images or image indices are saved in a database(such as databases 126) for later retrieval (block 1108). In the case ofgenerated template images, these images are run through the indexingphase 440 (see FIG. 4) to extract their image feature information (block1110). Block 1110 is optional if template image indices already exist,such template image indices being stored in the image index database301. One or more rule triggers corresponding to the template images orimage indices are also stored in a database. Rule triggers defineparticular conditions under which a specific template image or imageindex is used during the complement phase, as discussed in detail below.

At a block 1112, the complement module 320 is configured to obtainexpert knowledge data. Expert knowledge data comprises current fashiontrends (e.g., color blocking), what celebrities are wearing, whatdesigners are working on, reoccurring themes at fashion shows, colorscience, color charts, and other current color/texture combination datafrom expert sources. The expert knowledge data can be expressed inimages (e.g., photos of celebrities or runway models wearing certainoutfits and accessories) and/or non-pictorial form (e.g., text ofnoticed fashion trends, such as color blocking in general beingpopular). In the case of color science, for example, there may existscientifically based vocabulary of color schemes and themes. Additionaldetails regarding complementary color schemes and themes are provided athttp://www.dgp.toronto.edu/˜donovan/color/colorcomp.pdf andhttp://chenlab.ece.cornell.edu/people/congcong/publications/IJSTSP_Aesthetics.pdf.

Next at a block 1114, the complement module 320 determines orestablishes one or more rules based on the expert knowledge data. Thefashion trends may be distilled down to rules specifying which colorcombinations are pleasing, which accessories would go with particularoutfits, and the like. As an example, suppose a color scheme that iscurrently popular is a combination of colors A, B, and C. If a queryimage contains colors A and B in an appropriate mix, then a rule maydictate recommending an inventory image in a color C and which featuresan apparel piece that may be worn with and is different from the articleof clothing/accessory featured in the query image.

The rule(s) based on the expert knowledge data are used to generate oneor more template images (or template image indices) at block 1106. Thetemplate images or image indices are saved in a database along withcorresponding rule triggers at block 1108. If template image indicesexist instead of template images, then block 1110 may be omitted.Additional details provided above of blocks 1106-1110 in the context ofpurchasing behavior data are also applicable for expert knowledge data.In some embodiments, celebrity or runway model photos showing certainfashion trends obtained in block 1112 may comprise one or more of thetemplate images. Continuing the example of color combination A, B, and Cbeing a popular color scheme, a celebrity photo may show a celebritywearing an outfit and carrying accessories in colors A, B, and C. Thiscelebrity photo may be considered to be a template image and run throughthe indexing phase 440 in block 1110 to extract its image feature data(e.g., color distribution of colors A, B, and C). The corresponding ruletrigger for this template image may be if a query image contains colorA, B, or C above a pre-defined threshold amount of area, then use thistemplate image (or corresponding image index) to find matchingcomplementary items from among the inventory images. In otherembodiments, one or more template images or image indices may beconfigured based on the rules because there are no existing images thatembody the rules.

At a block 1116, the complement module 320 is configured to obtainsocial network data. Fashion is a social statement, which is influencedby the social circle of a given individual while that same individualalso influences those in his or her social circle. The complement module320 monitors and extracts fashion trend information from socialnetworks, and in particular, from social circles of users that have ormay submit a query image to the networked system 102. The informationpulled from social networks may include posted photos/images and textualcontent (such as a wish list, discussions of products, recent purchases,etc.).

Then at a block 1118, the complement module 320 determines orestablishes one or more rules based on the social network data. Thecomplement module 320 quantifies the choices and preferences of peoplein a user's social circle, and generates one or more rules based on thisdata. A different set of rules may exist for different groups of people,in which a group includes a user and members of his or her socialcircle. A set of rules for a given group of people may recommend, forexample, to a person within the group to buy a particular clothing itembecause it complements their friends' purchases. The field ofcollaborative filtering may be applicable, which deals with text andother transactions data (seehttp://en.wikipedia.org/wiki/Collaborative_filtering).

The rule(s) based on the social network data are used to generate one ormore template images (or template image indices) at block 1106. Thetemplate images or image indices are saved in a database along withcorresponding rule triggers at block 1108. If template image indicesexist instead of template images, then block 1110 may be omitted.Additional details provided above of blocks 1106-1110 in the context ofpurchasing behavior data and/or expert knowledge data are alsoapplicable for social network data. In some embodiments, photos/imagesposted on a user's or user's friends' social network page may comprise atemplate image. In other embodiments, one or more template images orimage indices may be configured based on the rules because there are noexisting images that embody the rules.

Although three recommendation learning sources —purchase behavior,expert knowledge, and social networks—are shown in FIG. 11, it isunderstood that fewer than three recommendation learning sources may beused to obtain the template image indices and corresponding ruletriggers. It is also understood that even if all three recommendationlearning sources are used to obtain respective template image indicesand corresponding rule triggers, that one or more of the recommendationlearning sources may be preferred in certain situations over others whenmaking recommendations to a use in response to a query image. The itemcategory for a given item may be provided by the seller when the item isbeing listed, may be automatically detected by a product identificationnumber corresponding to the item, or may be automatically detectedduring image processing.

With the template image indices and rule triggers established, acomplement phase commences to recommend one or more complementary itemsfrom among the listed items (or items offered for sale) in response to aquery image. As shown in FIG. 4, the operations of complement phase 444are similar to that of the matching phase 442 except as described below.It is understood that prior to commencement of the complement phase 444,indexing phase 440 occurred to index the inventory images as would bedone before matching phase 442.

In the complement phase 444, a user interfaces with the networked system102 via a website or an app installed on the device machine 110, 112 andprovides a query image to the networked system 102 (same as in thematching phase 442). The query image is received at block 402 b. Thequery image is processed to extract its image feature data in blocks 404b-408 b and 412 b-420 b as described above with respect to the matchingphase 442.

The operations of blocks 410 c, 424 c, 426 c, 428 c, and 430 ccorrespond respectively to operations of blocks 410 b, 424 b, 426 b, 428b, and 430 b except as follows:

-   -   Instead of comparing and calculating a similarity score between        each pair of the query image and a given inventory image, the        comparison and calculation of the similarity score occurs        between each pair of a template image and a given inventory        image.    -   The particular template image of the pairing is based on the        query image. The image feature data of the query image triggers        selection of a particular rule trigger. The particular rule        trigger, in turn, has a corresponding template image.    -   One or more different template images may be used depending on        the number of rule triggers that are applicable for the query        image.

In the matching phase 442, flow diagram 400 looks for inventory itemshaving the same or nearly the same image features as that of the queryimage. By contrast, in the complement phase 444, flow diagram 400 looksfor inventory items that are complementary to the query image. Thismeans that “ideal” inventory items should not have the same or nearlythe same image features as that of the query image. Instead, “ideal”inventory items are those items having the same or nearly the same imagefeatures as whatever an ideal complementary inventory item should havein accordance with the recommendation/complement rules established inFIG. 11. Template images serve as such ideal complementary inventoryitems; use of template images for comparison and similarity scoringagainst inventory images provides a mechanism to determine inventoryitems that are the most or nearly the most complementary to the queryimage.

Lastly, at block 422 c, the comparison module 316 providesrecommendation or complementary results to the device machine 110, 112for display. As with the matching phase 442, the inventory itemscomprising the recommendation/complementary results may be grouped byitem category and ordered from highest to lowest similarity score withineach group to facilitate orderly presentation on the device machine 110,112. The results are displayed on device machine 110, 112 similar tothat shown on screens 908, 910, 912 of FIG. 9B. The user can navigatewithin the recommendation/complementary results as he/she would with thematching results (e.g., similar to screens 914, 918, 920 of FIG. 9B). Insome embodiments, the colors and/or patterns of each of the itemsincluded in the complementary results are different from the colorsand/or patterns of the query image.

FIG. 12 illustrates an example (portion of the)recommendation/complementary results 1202, 1204 in response to a queryimage 1200. Notice that results 1202, 1204 can be different categoriesof items from that of the item featured in the query image 1200. Queryimage 1200 shows a dress. Results 1202. 1204 are shoes and handbag,respectively.

An alternative way to obtain the query image—other than the user takinga photo and uploading it to an e-commerce site or online marketplacehosted on the networked system 102 via an app or website on the devicemachine 110 or 112—is from websites not associated with the e-commerceor online marketplace. In fact, the websites can be competitor websites,online retailers' websites, fashion websites, social network websites,or any website comprising Hyper Text Markup Language (HTML)-based webpages (including HTML5-based web pages). Moreover, the matching and/orrecommendation results corresponding to the query image sourced from agiven website can be provided within the same given website. In otherwords, the matching and/or recommendation results need not be providedin a different website/web page from the given website that provided thequery image. These functionalities are facilitated by a browser plugininstalled on the device machine 110 or 112, as described in detailbelow.

FIG. 13 illustrates an example flow diagram 1300 for recommendinginventory items offered by an e-commerce or online marketplace atwebsites not associated with the inventory items' e-commerce or onlinemarketplace according to some embodiments. In situ acquisition of aquery image, presentation of inventory item recommendationscorresponding to the query image, and viewing of item detailscorresponding to one or more items included in the item recommendationsoccurs in a web page not associated with the inventory items' e-commerceor online marketplace.

At a block 1301, a browser plugin provided by the networked system 102(e.g., application servers 118) is installed on the device machine 110or 112. The browser plugin comprises a browser extension, jquery snippetof code, or browser plugin. The browser plugin can be a standalonemodule or part of an app, such as app 905 (FIG. 9B). The browser pluginincludes at least the browser plugin module 322.

Next at a block 1302, the browser plugin module 322 is configured tomonitor web browser activity at the device machine 110 or 112 to detecta request for a certain website (or web page). The request comprisesuser entry of a uniform resource locator (URL) address in a web browserincluded in the device machine 110 or 112, or the user clicking on ahyperlink to a web page. The certain website comprises a website (or webpage) from among a plurality of websites (e.g., competitor websites,online retailers' websites, fashion websites, social network websites,or any website comprising HTML-based web pages) from which a query imagecan be sourced and recommendation items can be presented to the userwithout leaving that website (or web page). Such a website (or web page)may also be referred to as a query image source or external third partyquery image source.

In response to detection of the request for a certain website (or webpage), the browser plugin module 322 is configured to modify the HTMLcontent of the web page before the web browser renders the web page onthe screen of the device machine 110 or 112 (block 1304). On-the-flychanges to HTML content is made using Document Object Model interface.The IMG HTML, tags are parsed. For example, the <img/> HTML tags can beparsed. If they are <img/> tags inside a <table/>, they are selectedover <img/> tags outside <tablet>. Also <img/> elements with a linkfollow through or call back is given higher priority than isolated<img/> elements. Modification to the requested web page includes, but isnot limited to, changing the callback or follow through link.

The modified web page is displayed on a screen of the device machine 110or 112 at a block 1306. At a block 1308, the browser plugin module 322and/or the user interest detection module 324 is configured to detectuser interest in or attention on an image included in the modified webpage. In one embodiment, user interest in an image is detected when theuser hovers a pointing device (e.g., mouse, trackpad, trackball, finger,etc.) at or near a particular image for at least a minimum time period.In another embodiment, user interest in an image is detected by trackingthe user's eye movement as he or she is viewing the web page. The humaneye rapidly scans interesting parts of a presented scene/content. Thismotion is called saccade. Eye tracking schemes are capable ofidentifying what part of a scene/content the eye is focusing on.Front-facing cameras included in smartphones and tablets may track eyemovement as the user views the web page on the device's screen. Inalternative embodiments, the browser plugin module 322 can provide oneor more graphical user interface (GUI) tools for the user to explicitlyspecify interest in a given image included on the web page. Example GUItook include, but is not limited to, a pointing tool, a highlightingtool, an area indictor tool, and the like. Whatever the particular wayto detect user interest in an image, the detection comprises identifyingthe user's spatial attention to a specific portion of the displayed webpage.

FIG. 14A illustrates an example screen 1400 displaying a (portion of a)web page 1401 of an online retailer (e.g., amazon.com). Web page 1401includes at least four images 1402, 1404, 1406, and 1408 of dressesoffered for sale by the online retailer. Although web page 1401 displaysprices of the four dresses, the user is unable to compare prices withother retailers' offerings without taking attention off of web page1401. FIG. 14B illustrates an example screen 1402 displaying the webpage 1401 in which the user expresses interest in image 1404. Inresponse to the detection, an image of interest indicator 1410 is shown(e.g., overlaid) on the web page 1401 corresponding to the image to besampled to find matches. Indicator 1410 may comprise a rectangularrubber band. Alternatively, indicator 1410 is optional and need not beshown to the user.

Once a user interested image is detected, the browser plugin module 322receives recommendation panel data corresponding to the user interestedimage at a block 1310. The user interested image (e.g., image 1404)comprises the query image that is automatically applied to the matchingphase 442 and/or the complement phase 444 (see FIG. 4) to present itemsoffered for sale by unrelated (and non-affiliated) e-commerce site oronline marketplace to the web page/website being displayed. The resultsof the matching and/or complement phases 442, 444 are included in therecommendation panel data. Al a block 1312, a recommendation panel (alsoreferred to as recommendations, item recommendations, matching items,complementary items, and the like) is rendered on the web page using therecommendation panel data received from the networked system 102. Therecommendation panel may comprise a floating panel, bar, or area thatcan be located at the bottom, top, side, middle, or any portion of thescreen. The recommendation panel may be fixed, floating, or movable onthe screen. The recommendation panel may be an opaque or translucentoverlay over a portion of the web page.

An example recommendation panel 1412 is shown at the bottom of thescreen 1402 in FIG. 14B. The recommendation panel 1412 includes a queryor sampled image 1414 that is the swatch portion of the image 1404.Image 1414 is optional. The recommendation panel 1412 also includesimages of the inventory items that best matches and/or complements theimage 1404. As an example, the top five, ten, or other pre-definednumber of best matches/complementary items can be displayed.

Next at a block 1314, the browser plugin module 322 and/or user interestdetection module 324 is configured to detect user interest in therecommendation panel. Similar to the discussion above for block 1308,user interest can be detected by mouse hover, eye gaze, or user explicitaction on or near the recommendation panel. In response to suchdetection, additional options or data for the recommendation panel isreceived from the networked system 102 (block 131) and displayed (block1318). FIG. 14C shows a screen 1420 displaying the web page 1401 and therecommendation panel 1412. The recommendation panel 1412 now includesadditional options or data 1422 showing more recommendation-relatedoptions to the user. For example, item category tabs are displayed, eachtab corresponding to top matching and/or complementary inventory itemsin the respective item category (e.g., tops and blouses, skirts,sweaters, handbags, shoes, etc.).

When the user expresses interest in one of the recommended items, suchas by moving a mouse over or pointing to a particular item image withinthe recommendation panel, the browser plugin module 322 and/or userinterest detection module 324 detects the user interest at a block 1320.In some embodiments, the image of the particular item will be enlargedin response, as shown in FIG. 14D. Image 1436 is larger than the otherimages of items being recommended in the recommendation panel 1434.Screen 1430 shown in FIG. 14D displays a web page 1432 from a socialnetwork site g., a Facebook page). An image 1433 included in the webpage 1432 is the user interested image (query image). Notice that webpage 1432 also includes advertisements 1438 or other images and/or text.Nevertheless, the recommendation panel 1434 does not present any itemsrelating to advertisements 1438. Image 1436 is also very similar to thedress in image 1433 except for different sleeves.

When the user clicks on (or otherwise explicitly selects) the particularitem image in the recommendation panel, item details corresponding tothe particular item are received at a block 1322 and displayed at ablock 1324. The item details are provided in a new web page dedicated tothe item. An item detail information page may be a web page within thee-commerce or online marketplace website. The user may also purchaseitems displayed in the recommendation panel.

FIG. 14E illustrates a screen 1440 showing a web page 1442 from anothersocial network site (e.g., Pinterest.com). An image 1444 posted on webpage 1442 is the query image. Correspondingly, a recommendation panel1446 is provided displaying top matching items available for sale by anunrelated e-commerce site or online marketplace. The first recommendeditem 1448 is identical to the dress that the user is interested in(e.g., image 1444).

FIG. 15 illustrates an example flow diagram 1500 showing operationsperformed by the networked system 102 (e.g., application server 118) tocalculate and provide recommendation data discussed in connection withFIGS. 13 and 14A-14E. At a block 1502, the networked system 102 receivesthe user interested image (or data equivalent) detected in block 1302(FIG. 13). In response, the matching phase 442 and/or the complementphase 444 is performed using the user interested image as the queryimage (block 1504). Next at a block 1506, the recommendation panel(data) is configured and provided to the device machine 110 or 112. Therecommendation panel data is received at the block 1310 (FIG. 13).

In response to detecting user interest in the recommendation panel,configure and provide additional options for the recommendation panel ata block 1508. The additional options are received at the block 1316(FIG. 13). In response to detecting user interest in a particular itemdisplayed in the recommendation panel, provide item details for theparticular item at a block 1510. Such item details are received at thedevice machine 110 or 112 at block 1322 (FIG. 13).

Thus, a mechanism is provided to attract potential customers from acompetitor's portal or a non-commerce portal (e.g., social networksites). Item information is also presented on an as needed basis basedon the user's current intent. In alternative embodiments, other contentincluded in a web page, such as MAIL tags, text surrounding an image,cookies, and/or user profile/account information, can augment the use ofan image from a web page as a query image to provide itemrecommendations based on user's attention on the web page image.

In this manner, image feature data extraction and uses of the imagefeature data are disclosed herein. In an indexing phase, inventoryimages corresponding to inventory items, user provided photo of an item,or an image included in a (non-affiliated) website or web page comprisesinput or sample images. In one embodiment, indexing is performed on theinventory images (for certain item categories such as clothing,accessories, bedding, and home furnishings) to extract their imagefeatures and attributes, and populate an image index database in whichthe inventory items are indexed by their image features and itemcategories. Extracted image features include, but are not limited to,color histogram, pattern, dominant color, and orientation histogram. Ina recommendation phase (matching phase and/or complement phase), theinformation in the image index database is accessed when a non-inventoryimage is received (also referred to as a query image) in order toprovide recommendations of inventory items corresponding to the queryimage. Recommendations (also referred to as recommendation results)comprise inventory items that best match the query image and/orinventory items that are best complements to the query image. In thecase of complementary matches, fashion trends and/or pleasingcolor/pattern combinations are gathered from one or more sources,complementary rules are distilled from the gathered data, templateimages or image indices and rule triggers are generated that embody orcorrespond to the complementary rules, and the templates images/imageindices are compared against inventory images to find complements to thequery image.

In another embodiment, user preference for certain of the items providedin the recommendation results is used to re-order or re-rank the listingorder of the items within the recommendation results. In still anotherembodiment, user's intent or interest is estimated while the user viewsa website/web page based on spatial attention detection. User's interestin an image included in the website/web page is used as the input,sample, or query image to return recommendation results corresponding tothat image (matching and/or complementary item recommendations). Therecommendation results are provided on the same website/web page withoutthe user having to turn attention away to another page. The user cannavigate within the recommendation results to view item details andpurchase an item included in the recommendation results.

It is contemplated that alternative embodiments for performing matchingand uses of the image feature data are possible. For example:

-   -   Apply different bin sizes of histograms for the histogram        matching scheme. And/or use different distance functions to        figure out which distance function works best in a        class-specific sense. The distance metrics in a data-driven        manner that minimizes the distance between similar items while        maximizing that between dissimilar ones.    -   Use different local shape based features to determine which        features are best suited for which item class. Furthermore, the        local shape based features are selected for particular item        categories that specifically work well for them.    -   Automatically detecting logos of different manufacturers on        their product. Different detection methods include, but not        limited to, template based matching, local shape detector based        matching, and ensemble based matching methods.    -   Many products can be uniquely identified based on the text that        might be present on them. Thus, include an optical character        recognition engine designed to detect text and other        alpha-numeric characters on a product image.    -   Personalization of search results—Instead of returning the same        results for a particular query image independent of the user,        make search process more personalized, and return search results        that best suites the taste of the user interested in a        particular query. This would require some form of result        re-ranking done in an active learning setup.    -   Detecting the style of a particular query image is relevant for        generating good similarity results, particularly for a fashion        app. Perform some basic style detection automatically, so that        the user is presented with the most likely categories from which        the query object might belong to. Apply different classification        and object recognition schemes, possibly different schemes for        different item categories.    -   Here the idea is that the user gets a list of results. By using        the gyroscope feature of the mobile device (e.g., smartphone or        tablet), a new set of items from the ones shown can be provided        in response to detection of a shaking motion. Additionally, by        using the dimensions of the shake along different axes—X        direction, Y direction, or Z direction—re-sorts can be performed        along a predetermined respective dimension like color, texture,        or shape.    -   Suppose a user finds an everyday item with pleasing colors        and/or patterns. Examples of these everyday items could be a        painting, a car, or even a pet. She wants to know if there are        clothes at a given e-commerce site or online marketplace that        appear similar to the everyday item of interest. She takes a        picture of the item of interest using a smart phone, and uses        the app (e.g., app 905) provided by the given e-commerce site or        online marketplace to automatically obtain items that look        similar to the everyday item. These results are organized        according to their respective categories, and the user can        choose a particular category to explore the matching items of        different classes of objects.    -   This use case is for the situation where a user has a clothing        apparel (e.g. a skirt or a top), and wants to find complementing        accessories for that clothing item (e.g., a matching scarf, or a        hat). She takes a picture of the item of the existing clothing        apparel, and uses the app (e.g., app 905) to search what items        are available on the given e-commerce site or online marketplace        that look similar to the existing clothing apparel. Based on the        similarity between the visual features of the input image and        the inventory items, the app returns items matching the existing        clothing apparel. The user can choose a particular category to        explore the matching items of different accessory categories.    -   Consider the scenario where a user visits a clothing store, and        finds an interesting item. She wants to check prices of similar        looking items at a given e-commerce site or online marketplace.        She uses a mobile device (e.g., smartphone) to take a photo of        the interesting item in the store and launch the app (e.g., app        905) to view items available at the given e-commerce site or        online marketplace that look similar to the item of interest in        the clothing store.    -   Celebrity's choice of clothing is usually considered a fashion        statement, and many people want to dress like celebrities. This        use case is for situations where a user wants to dress like a        celebrity. The user can either take a picture of an existing        image where the celebrity is wearing the dress of interest, or        if they have the image already in the phone, they can upload it        directly. Based on the similarity between the visual features of        the input image and the inventory items, the app (e.g., app 905)        returns items matching the celebrity's clothes. The user can        choose a particular category to explore the similar items of        different clothing categories. This use case can be also used        with an automatic face recognition technology, which would        recognize the face of the celebrity, and show results that match        the flavor of the celebrity's style.    -   Consider a scenario where a seller wants to sell their clothes        in an online marketplace. To make the process of filling out the        details of the clothes more efficient, the seller uploads an        image of the piece of clothing to be sold. The app (e.g., app        905) finds the best matches for this item based on visual        similarity, and shows the results to the seller. The seller can        now select the item that has the exact match with the item to be        sold. Given this information, the system can auto-fill most of        the fields for the seller to make the process of transferring        information about the item to be sold more efficient.

FIG. 16 shows a diagrammatic representation of a machine in the exampleform of a computer system 1600 within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed. The computer system 1600 comprises,for example, any of the device machine 110, device machine 112,applications servers 118, API server 114, web server 116, databaseservers 124, or third party server 130. In alternative embodiments, themachine operates as a standalone device or may be connected (e.g.,networked) to other machines. In a networked deployment, the machine mayoperate in the capacity of a server or a device machine in server-clientnetwork environment, or as a peer machine in a peer-to-peer (ordistributed) network environment. The machine may be a server computer,a client computer, a personal computer (PC), a tablet, a set-top box(STB), a Personal Digital Assistant (PDA), a smart phone, a cellulartelephone, a web appliance, a network router, switch or bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example computer system 1600 includes a processor 1602 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU), orboth), a main memory 1604 and a static memory 1606, which communicatewith each other via a bus 1608. The computer system 1600 may furtherinclude a video display unit 1610 (e.g., liquid crystal display (LCD),organic light emitting diode (OLED), touch screen, or a cathode ray tube(CRT)). The computer system 1600 also includes an alphanumeric inputdevice 1612 (e.g., a physical or virtual keyboard), a cursor controldevice 1614 (e.g., a mouse, a touch screen, a touchpad, a trackball, atrackpad), a disk drive unit 1616, a signal generation device 1618(e.g., a speaker) and a network interface device 1620.

The disk drive unit 1616 includes a machine-readable medium 1622 onwhich is stored one or more sets of instructions 1624 (e.g., software)embodying any one or more of the methodologies or functions describedherein. The instructions 1624 may also reside, completely or at leastpartially, within the main memory 1604 and/or within the processor 1602during execution thereof by the computer system 1600, the main memory1604 and the processor 1602 also constituting machine-readable media.

The instructions 1624 may further be transmitted or received over anetwork 1626 via the network interface device 1620.

While the machine-readable medium 1622 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding or carrying a set of instructions for execution by themachine and that cause the machine to perform any one or more of themethodologies of the present invention. The term “machine-readablemedium” shall accordingly be taken to include, but not be limited to,solid-state memories, optical and magnetic media, and carrier wavesignals.

It will be appreciated that, for clarity purposes, the above descriptiondescribes some embodiments with reference to different functional unitsor processors. However, it will be apparent that any suitabledistribution of functionality between different functional units,processors or domains may be used without detracting from the invention.For example, functionality illustrated to be performed by separateprocessors or controllers may be performed by the same processor orcontroller. Hence, references to specific functional units are only tobe seen as references to suitable means for providing the describedfunctionality, rather than indicative of a strict logical or physicalstructure or organization.

Certain embodiments described herein may be implemented as logic or anumber of modules, engines, components, or mechanisms. A module, engine,logic, component, or mechanism (collectively referred to as a “module”)may be a tangible unit capable of performing certain operations andconfigured or arranged in a certain manner. In certain exampleembodiments, one or more computer systems (e.g., a standalone, client,or server computer system) or one or more components of a computersystem (e.g., a processor or a group of processors) may be configured bysoftware (e.g., an application or application portion) or firmware (notethat software and firmware can generally be used interchangeably hereinas is known by a skilled artisan) as a module that operates to performcertain operations described herein.

In various embodiments, a module may be implemented mechanically orelectronically. For example, a module may comprise dedicated circuitryor logic that is permanently configured (e.g., within a special-purposeprocessor, application specific integrated circuit (ASIC), or array) toperform certain operations. A module may also comprise programmablelogic or circuitry (e.g., as encompassed within a general-purposeprocessor or other programmable processor) that is temporarilyconfigured by software or firmware to perform certain operations. Itwill be appreciated that a decision to implement a module mechanically,in dedicated and permanently configured circuitry, or in temporarilyconfigured circuitry (e.g., configured by software) may be driven by,for example, cost, time, energy-usage, and package size considerations.

Accordingly, the term “module” should be understood to encompass atangible entity, be that an entity that is physically constructed,permanently configured (e.g., hardwired), non-transitory, or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich modules or components are temporarily configured (e.g.,programmed), each of the modules or components need not be configured orinstantiated at any one instance in time. For example, where the modulesor components comprise a general-purpose processor configured usingsoftware, the general-purpose processor may be configured as respectivedifferent modules at different times. Software may accordingly configurethe processor to constitute a particular module at one instance of timeand to constitute a different module at a different instance of time.

Modules can provide information to, and receive information from, othermodules. Accordingly, the described modules may be regarded as beingcommunicatively coupled. Where multiples of such modules existcontemporaneously, communications may be achieved through signaltransmission (e.g., over appropriate circuits and buses) that connectthe modules. In embodiments in which multiple modules are configured orinstantiated at different times, communications between such modules maybe achieved, for example, through the storage and retrieval ofinformation in memory structures to which the multiple modules haveaccess. For example, one module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further module may then, at a later time,access the memory device to retrieve and process the stored output.Modules may also initiate communications with input or output devicesand can operate on a resource (e.g., a collection of information).

Although the present invention has been described in connection withsome embodiments, it is not intended to be limited to the specific formset forth herein. One skilled in the art would recognize that variousfeatures of the described embodiments may be combined in accordance withthe invention. Moreover, it will be appreciated that variousmodifications and alterations may be made by those skilled in the artwithout departing from the scope of the invention.

The Abstract is provided to allow the reader to quickly ascertain thenature of the technical disclosure. It is submitted with theunderstanding that it will not be used to interpret or limit the scopeor meaning of the claims. In addition, in the foregoing DetailedDescription, it can be seen that various features are grouped togetherin a single embodiment for the purpose of streamlining the disclosure.This method of disclosure is not to be interpreted as reflecting anintention that the claimed embodiments require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle disclosed embodiment. Thus the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment.

What is claimed is:
 1. A method comprising: receiving, by a networkedsystem, a query image from a remote device; performing, by the networkedsystem, image processing to extract an image attribute of the queryimage; comparing, using a processor of the networked system, theextracted image attribute of the query image to an attribute associatedwith each of a plurality of items to identify items similar to the queryimage; causing display of a plurality of item recommendations comprisingat least a portion of the identified items on a user interface of theremote device, the plurality of item recommendations being displayedbased on similarity between the extracted image attribute of the queryimage and the attribute associated with each of the identified items;receiving, by the networked system from the remote device, a selectionof a recommendation from the plurality of item recommendations; inresponse to receiving the selection of the recommendation, determining,by the networked system, a set of updated recommendations based on theselected recommendation; and causing display of item details of theselected recommendation and the set of updated recommendations on theremote device.
 2. The method of claim 1, wherein the determining the setof updated recommendations comprises including at least one of theplurality of item recommendations not previously selected.
 3. The methodof claim 1, wherein the comparing the extracted image attribute of thequery image to the attribute of each of the plurality of items comprisescomparing a dominate color of the query image to a dominate color ofeach of the plurality of items.
 4. The method of claim 1, wherein thecomparing the extracted image attribute of the query image to theattribute of each of the plurality of items comprises comparing apattern of the query image to a pattern of each of the plurality ofitems.
 5. The method of claim 1, wherein the query image comprises animage captured by an application on the remote device.
 6. The method ofclaim 1, wherein the causing display of the item details of the selectedrecommendation comprises causing display of an image of the selectedrecommendation in a larger size than when the image was displayed withthe plurality of item recommendations.
 7. The method of claim 1, furthercomprising providing an application for installation on the remotedevice, the application including camera capabilities that configure theremote device to capture the query image, the application transmittingthe query image to the networked system.
 8. The method of claim 1,wherein: the performing the image processing to extract the imageattribute of the query image comprises automatically detecting a logo ofa manufacturer; and the comparing the extracted image attribute of thequery image to an attribute of each of a plurality of items to identifyitems similar to the query image comprises identifying items from thesame manufacturer.
 9. The method of claim 1, wherein: the performing theimage processing to extract the image attribute of the query imagecomprises using an optical character recognition engine to detect textor alpha-numeric characters; and the comparing the extracted imageattribute of the query image to an attribute of each of a plurality ofitems to identify items similar to the query image comprises identifyingitems related to the text or alpha-numeric characters.
 10. The method ofclaim 1, wherein the query image depicts an article of clothing or aportion of the article of clothing.
 11. A system comprising: one or morehardware processors; and a storage device storing instructions that,when executed by the one or more hardware processors, causes the one ormore hardware processors to perform operations comprising: receiving aquery image from a remote device; performing image processing to extractan image attribute of the query image; comparing the extracted imageattribute of the query image to an attribute associated with each of aplurality of items to identify items similar to the query image; causingdisplay of a plurality of item recommendations comprising at least aportion of the identified items on a user interface of the remotedevice, the plurality of item recommendations being displayed based onsimilarity between the extracted image attribute of the query image andthe attribute associated with each of the identified items; receiving,from the remote device, a selection of a recommendation from theplurality of item recommendations; in response to receiving theselection of the recommendation, determining a set of updatedrecommendations based on the selected recommendation; and causingdisplay of item details of the selected recommendation and the set ofupdated recommendations on the remote device.
 12. The system of claim11, wherein the determining the set of updated recommendations comprisesincluding at least one of the plurality of item recommendations notpreviously selected.
 13. The system of claim 11, wherein the comparingthe extracted image attribute of the query image to the attribute ofeach of the plurality of items comprises comparing a dominate color ofthe query image to a dominate color of each of the plurality of items.14. The system of claim 11, wherein the comparing the extracted imageattribute of the query image to the attribute of each of the pluralityof items comprises comparing a pattern of the query image to a patternof each of the plurality of items.
 15. The system of claim 11, whereinthe query image comprises an image captured by an application on theremote device.
 16. The system of claim 11, wherein the causing displayof the item details of the selected recommendation comprises causingdisplay of an image of the selected recommendation in a larger size thanwhen the image was displayed with the plurality of item recommendations.17. The system of claim 11, wherein the operations further compriseproviding an application for installation on the remote device, theapplication including camera capabilities that configure the remotedevice to capture the query image, the application transmitting thequery image to the networked system.
 18. The system of claim 11,wherein: the performing the image processing to extract the imageattribute of the query image comprises automatically detecting a logo ofa manufacturer; and the comparing the extracted image attribute of thequery image to an attribute of each of a plurality of items to identifyitems similar to the query image comprises identifying items from thesame manufacturer.
 19. The system of claim 11, wherein: the performingthe image processing to extract the image attribute of the query imagecomprises using an optical character recognition engine to detect textor alpha-numeric characters; and the comparing the extracted imageattribute of the query image to an attribute of each of a plurality ofitems to identify items similar to the query image comprises identifyingitems related to the text or alpha-numeric characters.
 20. Amachine-readable storage medium storing instructions that, when executedby the one or more hardware processors, causes the one or more hardwareprocessors to perform operations comprising: receiving a query imagefrom a remote device; performing image processing to extract an imageattribute of the query image; comparing the extracted image attribute ofthe query image to an attribute associated with each of a plurality ofitems to identify items similar to the query image; causing display of aplurality of item recommendations comprising at least a portion of theidentified items on a user interface of the remote device, the pluralityof item recommendations being displayed based on similarity between theextracted image attribute of the query image and the attributeassociated with each of the identified items; receiving, from the remotedevice, a selection of a recommendation from the plurality of itemrecommendations; in response to receiving the selection of therecommendation, determining a set of updated recommendations based onthe selected recommendation; and causing display of item details of theselected recommendation and the set of updated recommendations on theremote device.